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Method for remote control, system using the same method, and computer 
program product performing the same method 

FIELD OF THE INVENTION 

The present invention relates to a remote control data processing 
method in a remote control system including a client computer, a host computer, 
and a network for linking them, a^system using the method thoroof and a 
computer program product performing the method thoroof . 



10 BACKGROUND OF THE INVENTION 

As Recently, ao the communication networks have recently network io 
expanded and computer systems mounting easy-to-use graphical user interface 
are developed, remote control systems for controlling electronic appliances at 
distant places through the network are being widely distributed distributing . 
15 One of the problems of remote control by network is that it takes a 

certain time to transmit a remote control signal over fe g-a long distance. As a 
result, a time delay occurs between the local operation and operation at the 
distant place. 

The delay time in the network becomes a serious problem in the case of 
20 composite operations which are composed of plural operations. In such 
composite operations, the interval between ef-one operation and another ether 
is an important factor for judging what operation has been done. For example, 
suppose to opon a computer program is opened by double-clicking an icon on the 
screen of a computer in a distant place. 
25 An example of a problem in remote control which is caused by a_delay 

time in the network is explained below. 

Fig. 8 is a configuration of a remote control system. 
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In Fig. 8, the remote control system contains a client computer 2, a host 
computer 1 which is controlled by the client computer 2 from a distant place, 
and a network 3 Unking these computers. 

The host computer 1 comprises an information processing unit 11, a 
display unit 12, a communication unit 13, and a pointing device 14. The 
display unit 12 displays the image information that is p rocessed in the 
information processing unit 11. The communication unit 13 exchanges 
information with the network 3. The pointing device 14 manipulates the host 
computer 1. 

10 The client computer 2 comprises an information processing unit 21, a 

display unit 22, a communication unit 23, and a pointing device 924. The 
display unit 22 displays the image information that is p rocessed in the 
information processing unit 21. The communication unit 23 exchanges 
information with the network 3. The pointing device 9 24-manipulates the 
15 client computer 2. 

The pointing devices 14, 9_24~are a mouse, a digitizer, and other input 

devices. 

The pointing devices 14, 9 24~have on/off buttons for moving the pointer 
so as to indicate for indicating a position on the screen of the respective display 
20 units 12, 22, and to select oolccting the object that is indicated by the pointer by 
inputting coordinates data. 

The user can click the on/off button twice quickly without moving the 
pointing device (without updating the device coordinates data), which is called a 
double click. In this case, if the interval between the first click and second 
25 click is long, this operation is not recognized as a double click. 

The network 3 is a wired or wireless network such as a local area 
network (LAN) or a wide area network (WAN). 
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The operation of the remote control system will now be ie-explained. 
The host computer (hereinafter called host PC) 1 transmits the image 
information that is output in the display unit 12 to the communication unit 23 
of the client computer (hereinafter called client PC) 2 from the communication 
unit 13 through the network 3. At this time, the client computer 2 processes 
the image information that is sent from the host computer 1 in the information 
processing unit 21, and outputs the processed image information to the display 
unit 22. 

When the pointing device 9_34-is manipulated in the client PC 2, this 
10 operation information is sent to the communication unit 13 from the 
communication unit 23 through the network 3 . The host PC 1 processes the 
operation information that is sent from the client PC 2 as the operation 
information by manipulation of the pointing device 14. 

The host PC 1, when the image information that is output in the display 
15 unit 12 is changed by manipulation of the pointing device 14, sends the 
differential image information before and after the change from the feet 
communication unit 13 to the communication unit 23 of the client PC 2. 

The client PC 2 updates the image information in the display unit 22 
according to the differential image information that is sent from the host PC 1. 
20 In this manner, the host PC 1 which is located in a distant place can be 

operated from the client PC 2, and the same image information as in the 
display unit 12 of the host PC 1 can be shown in the display unit 22 of the client 
PC 2. 

In this remote control system, if the delay time in the network is large, 
25 the following phenomena may occur. 

i) If the double click which is entered in the client PC 2 is transmitted 
to the host PC 1 as operation information, it is not processed as a double click in 
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the host PC 1. 

ii) Two single clicks which are entered in the client PC 2 are processed 
as a double click at the host PC 1. 

Such incorrect processes may occur if the time interval between packets 
is changed by an irregular delay characteristic of the network while the data of 
two packets showing two successive presses on the button of the pointing device 
9_24-flow on the network. At this time, the host PC 1 recognizes and processes 
as if the button was pressed at a different timing from the operation in 
transmission. 

Moreover, as compared with the case of the user manipulating the host 
PC 1 directly, when in caoo that the host PC 1 is_remote-controled by the client 
PC 2, it takes time until the input result at the client PC 2 is reflected in the 
first display unit 12 of the host side. Therefore, the user of the client computer 
2 takes time until knowing that the double click is correctly entered at the host 
PC 1. As a result, the-working efficiency decreases dropo . 



SUMMARY OF THE INVENTION 

It is therefore hcnoc an object of the invention to present a method, 
system and a computer program product for processing remote control data 
20 even if the delay time in the network is large. In the present invention, the 
host computer terminal and client computer terminal are coupled through the 
network, and the remote control system is built up. 
The client computer terminal comprises,- 

an information processing unit,_- 

25 a communication unit for exchanging information with the network, 



and - 



-a pointing device for controlling the client computer terminal. 



4 



In this client computer terminal, the information processing unit 
transforms the operating procedure of the pointing device into the operation 
information including the button state data, coordinates data, and time interval 
data between the operations, and the communication unit sends the 
information to the network. 

On the other hand, the host computer terminal comprises an 
information processing unit and a communication unit which are coupled with 
the network. In the host computer terminal, the information processing unit 
decomposes the operation information that is received in its communication 
10 unit into the operating procedure at the client terminal, and reproduces the 
operating procedure. 

In this manner, the operating procedure of the pointing device at the 
client terminal can be reproduced at the host terminal without having the 
effects of an irregular delay of the network. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows a configuration of a remote control system in an 
embodiment of the present invention. 

Fig. 2A is a functional block diagram of an information processing unit 
of aclient PC in the embodiment of the present invention. 

Fig. 2B is a functional block diagram of an information processing unit 
of ajiost PC in the embodiment of the present invention. 

Fig. 3 is a flowchart showing the first button ON judging process in a 
double click input of the client PC in the embodiment of the present invention. 

Fig. 4 is a flowchart showing the first button OFF judging process after 
the first button is clicked ON in a double click input of the client PC in the 
embodiment of the present invention. 
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Fig. 5 is a flowchart showing the second button ON judging process 
after the first button is clicked OFF in a double click input of the client PC in 
the embodiment of the present invention. 

Fig. 6 is a flowchart showing the process until advancing to the first 
button ON judging process in a double click input after the double click input or 
after going out of the double click input condition of the client PC in the 
embodiment of the present invention. 

Fig. 7 is a flowchart showing ajreception process of an operation packet 
related to a pointing device of the client PC in the host PC in the embodiment of 
the present invention. 

Fig. 8 shows a configuration of a conventional remote control system. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMEN T OF THE 

INVENTION 

An embodiment of the present invention is explained with reference by 
referring to Figs. Fig. 1 to Fig. 7. 

A configuration of the remote control system in the embodiment is 
explained with reference by referring to Fig. 1 and Fig. 2. 

Fig. 3 to Fig. 6 are flowcharts showing the transmission process of an 
operation packet about a pointing device 44 of the client computer in the 
embodiment. 

In Fig. 1, information processing units 31, 41 monitor the operation of 
pointing devices 34, 44 of a host computer (host PC) 30 and a client computer 
(client PC) 40 . respectively. 

As the pointing device is manipulated, when the indicated coordinates 
are moved or the on/off of the button is changed, the processing units 31, 41 
generate data, and display the coordinates data and the button status data on 



the screen of the display units 32, 42 *mit-as a pointe r, respectively . The 
processing units 31, 41 form data into packets, and send the data to the 
counterpart information processing unit 31, 41 of the partner of communication 
through communication units 33, 43 and ^network 70. 
5 When the button of the pointing device is turned on, off and on within 

a maximum click interval Tl without a^change in the coordinates that are 
indicated by the pointing device, the processing units 31, 41 judge an input of 
double click. 

The processing units 31, 41 have a timer for measuring the maximum 
10 click interval Tl. 

The timer has a function like a watchdog timer, and measures the time 
from a first click to a second click on the pointing device. The timer setting 
time Tl is counted from 0 seconds oocond , and is the duration from the start of 
an operation at an arbitrary timing until the en d of the operation . 
15 Fig. 2A is a functional block diagram showing the operation of the 

information processing unit 41 of the client PC 40. 

Fig. 2B is a functional block diagram showing the operation of the 
information processing unit 31 of the host PC 30. 

In Fig. 2A, a timer 60 counts the time as the reference for detecting the 
20 operation of the pointing device 44. An operation determining unit 62 detects 
and judges the move of the pointing device 44 and an on/off operation of the i ts 
button of the pointing device 44 . Based on a On the baoio of result of this 
detection and judgement, the operation determining unit 62 further generates 
operation information. This operation information includes the coordinates 
25 data showing the move of the pointing device 44, button status data showing 
the on/off operation of the button, and the time between the operations. An 
informing unit 64 outputs data for informing the user of the operation of the 
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pointing device 44 . A transmission data output unit 66 outputs the data 
regarding about the operation of the pointing device^ including the operation 
information^ to the network 70 through the communication unit 43. 

In Fig. 2B, a timer 50 counts the time for judging a double click of the 
pointing device 34 at the host PC 30. A watching unit 52 monitors the 
reception of data regarding about the operation of the pointing device 44 
including the operation information that is sent from the client PC 40. A data 
determining unit 54 judges the operation of the pointing device 44^at the client 
PC 40 based on the data including the operation information that is sent from 
the client PC 40. A reproducing unit 56 reproduces the operation of the 
pointing device 44 at the client PC 40 depending on the data that is j udged by 
the data determining unit 54 at the host PC 30. 

The operation in the client PC 40 will now be is— explained with 
reference by roforring to Figs. 3-6 Fig. 3 to Fig. 6 . 

The remote control data processing in the client PC 40 includes the 
following processing in the input of a_double click. 

1) Judging the first button to be ON, 

2) Judging the first button to be OFF after judging the first button to 

be_ON, 

3) Judging the second button to be ON after judging the first button to 
be_OFF, and 

4) Processing before advancing to the first button ON judging process 
in a double click input, after the double click input or after going out of the 
double click input condition. 

The procedure will now be ie-explained sequentially from the flowchart 
in Fig. 3. 

Fig. 3 is a flowchart showing the first button ON judging process in a 
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double click input in the client PC 40. 

In Fig. 3, at the click interval acquiring step S101, the timer 60 acquires 
the maximum click interval Tl for judging a double click from the host PC 30. 

At the first watching step S102, the operation determining unit 62 
judges whether or not if-the client side pointing device 44 has been operated-er 
net. If the pointing device 44 has not been operated, the watching step 
continues until the operation is recognized. When the pointing device 44 is 
operated, the process goes to the first move judging step S103. 

At step S103, the operation determining unit 62 judges whether or not 
if-the operation of the pointing device 44 is accompanied by a move or not . If 
the pointing device 44 has not been moved, it is judged regarded that the button 
of the pointing device 44 is turned on, and the process goes to the first timer 
start step S104. If the pointing device 44 has been moved, the process goes to 
the first operation packet transmit step S105. 



At step SI 04, the timer 60 starts, and the flowchart shown in Fig. 4 is 

started. 

At step S105, the transmission data output unit 66 sends a coordinates 
data packet showing the move operation that is generated in the operation 
determining unit 62 to the host PC 30. After transmission of the coordinates 
data packet , the process goes to the first ON judging step S106. 

At step S106, the determining unit 62 judges whether or not tf -the 
button of the pointing device 44 has been turned on or not . 

If the button is turned on, the flowchart shown in Fig. 6 is started. 
If the button is not turned on, going back to step S102, the same process 
25 is repeated. 

The flowchart shown in Fig. 4 will now be io-explained. 
Fig. 4 is a flowchart showing the first button OFF judging process after 
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the first button is judged to be ON in ajiouble click input in the client computer 
40. 

At the first lapse time measuring step S201, the timer 60 measures the 
lapse time, and judges if the time has exceeded Tl or not. When the lapse time 
exceeds exceeding Tl, the process goes to the first timer stop step S204. 
Otherwise, the process goes to the second watching step S202. 

At step 202, the determining unit 62 judges whether or not if-the 
pointing device 44 is operate d or not . If the pointing device 44 is judged not to 
be operated, going back to step S201, the timer continues to watch until the 
lapse time exceeds cxcocding time TL 

When the pointing device is judged to be operated, the process goes to 
the second move judging step S203. 

At step S203, it is judged if the operation of the pointing device 44 is 
accompanied by a move or not. 

If the pointing device 44 is not accompanied by a moved, the 
determining unit 62 judges that the device 44 is turned off, and the flowchart in 
Fig. 5 is started. 

On the other hand, if the pointing device 44 is judged to have been 
accompanied by a f f-moved, it is regarded that the pointing device 44 is moved 
with the button remaining in the ON position, and the process goes to the 
second timer stop step S206. 

At step S206, the timer 60 stops, and the transmission data output unit 
66 transmits the packet data showing the button ON to the host PC 30 through 
the communication unit 43 (the second ON packet transmit step S207). 

In succession, the coordinates data packet of the pointing device 44 is 
sent to the host PC 30 (the second operation packet send step S208). 

After transmissio n of the coordinates data packet of the pointing device 
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44, the process goes to the first OFF judging step S209. 

At step S209, the determining unit 62 judges whether or not if-the ON 
state button is manipulated to the OFF state or not . 

If the button is manipulated to the OFF state, going back to step S102 
in Fig. 1, the same process is repeated. 

When the button remains in the ON position, the flowchart in Fig. 6 is 

started. 

At the first timer stop step S204, the timer 60 stops, and the process 
goes to first ON packet send step S205. 

At step S205, the output unit 66 transmits the button ON packet data 
to the host PC 30, and the flowchart in Fig. 6 is started. 

The steps shown in the flowchart in Fig. 5 will now be a re-explained. 

Fig. 5 is a flowchart showing the second button ON judging process 
after judging the first button to be OFF in a double click input in the client PC 
40. 

At the second lapse time measuring step S301, the timer 60 measures 
the lapse time, and judges whether or not i f-the time has exceeded Tl or not . 
When the lapse time exceeds exceeding Tl, the process goes to the fourth timer 
stop step S307. Otherwise, the process goes to the third watching step S302. 

At step 302, the determining unit 62 judges whether or not i f-the second 
pointing device 44 is operated or not . If the pointing device 44 is not operated, 
going back to step S301, the timer continues to watch until the lapse time 
exceeds exceeding time Tl. When operated, the process goes to the third timer 
stop step S303. 

At step S303, the timer 60 stops, and the process goes to the third move 
judging step S304. 

At step S304, the determining unit 62 judges whether or not i £-the 
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operation of the pointing device 44 of the client PC 40 is accompanied by a move 
or not . 

If the pointing device 44 has not been accompanied by a move *fteved, 
the determining unit 62 judges that the device 44 is turned off. At this time, 
the output unit 66 transmits an operation command showing a double click to 
the host PC 30 (double click send step S305). 

Consequently, the informing unit 64 transmits data to an audio output 
unit or a video output unit of the client PC 40 so as to inform the user of the 
transmission of a^double click command by voice or display (informing step 
S306). 

The next process goes to step S401 in Fig. 6. 

If the pointing device 44 is moved by an operation, on the other hand, 
the determining unit 62 judges that the button is turned on (step S102), and 
then turned off (step S202), and consequently that the pointing device 44 is 
moved. Therefore, first, the transmission data output unit 66 first transmits 
packet data showing the button to be ON to the host PC 30 (fourth ON packet 
send step S310). 

Next, the transmission data output unit 66 transmits packet data 
showing the button to be OFF to the host computer 30 (fourth OFF packet send 
step S311). 

Further, the transmission data output unit 66 transmits the coordinates 
data showing the coordinates of the moved position of the pointing device 44 to 
the host computer 30 (third operation packet send step S3 12). Then A the 
process goes to the third ON judging step S313. 

At the second lapse time measuring step S301, when the lapse time of 
the timer 60 exceeds Tl, the process goes to the fourth timer stop step S307. 

At step S307, the timer 60 stops, and the process goes to the third ON 
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packet transmit step S308. 

At step S308, the transmission data output unit 66 transmits the 
packet data showing a_en-button to be ON to the host PC 30, and the process 
goes to third OFF packet send step S309. 

At step S309, the transmission data output unit 66 transmits the 
packet data showing a_an-button to be OFF to the host PC 30, and the process 
returns to step S102 in Fig. 1. 

At the third ON judging step S3 13, the operation determining unit 62 
judges whether or not i£the operation judged at step S302 is an ON operation of 
10 the button-or^et. 

In the case of an ON operation of the button, the process goes to the 
fourth watching step S401 in Fig. 6. 

Otherwise, the process returns to step S102 in Fig. 1. 
The flowchart shown in Fig. 6 will now be i s-explained. 
15 Fig. 6 is a flowchart showing the process until advancing to the first 

button ON judging process in a double click input after the double click input or 
after going out of the double click input condition in the client PC 40. 

At the fourth watching step S401, the determining unit 62 judges 
whether or not if-the second pointing device 44 is operate d or not . If the 
20 pointing device 44 is not operated, watching continues until an operation of the 
pointing device 44 is recognized. 

When the pointing device 44 is operated, the process goes to the fourth 
operation packet send step S402. 

At step S402, the transmission data output unit 66 transmits the 
25 operation packet showing the operation judged at step S401 to the host 
computer 1, and the process goes to the fourth OFF judging step S403. 

At step S403, the determining unit 62 judges whether or not i f-the 
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operation that is j udged at step S401 is an OFF operation of the button-eraet. 

In the case of the OFF operation of the button, the process returns to 
step S102 in Fig. 1. 

Otherwise, the process returns to step S401. 

Thus, in the information processing unit 41 of the client PC 40 , the 
operation determining unit 62 judges the operation of the pointing device 44, 
and generates operation information as packet data. The data output unit 66 
outputs the operation information as packet data to the network 70 through the 
communication unit 43. 

The processing Process of remote control data in the host PC 30 will 
now be is-explained-belew. 

Fig. 7 is a flowchart showing the reception process of an operation 
packet, that is, operation information regarding a bo**t-the pointing device 44 of 
the client PC 40 in the host PC 30 in the embodiment. 

Herein, T2 is the maximum click interval for judging a double click by 
the button ON-QFF-QN button operation of the pointing device 34 in the host 
PC 30. The timer 50 stops automatically when exceeding the time T2. 

In Fig. 7, at a reception watching step S501, the watching unit 52 
judges whether or not if-the operation packet data that is transmitted from the 
client computer 40 is received or not . 

If the operation packet data is not received, the watching unit 52 
continues to monitor until the reception of the operation packet data is 
confirmed. 

When the operation packet data is received, the process goes to the 
fourth move judging step S502. 

At step S502, the data determining unit 54 judges whether or not i f-the 
operation packet data that is received at step S501 shows the operation as 
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being accompanied by a^move of the pointing device 44 of the coordinates-ernset. 

If accompanied by a_move of the coordinates, the process goes to the 
fourth timer stop step S503. At this step, the timer 50 stops, and the process 
goes to the double click judging step S504. 

If not accompanied by a_move, the process goes to step S504. 
At step S504, the data determining unit 54 judges whether or not i f-the 
operation packet data that is received at step S501 shows a double click or not . 

If the received operation packet data shows a double click, the process 
goes to the first timer operation watching step S505. 
10 If the received operation packet data does not show a double click, the 

process goes to the button operation judging step S509. 

At the first timer operation watching step S505, the watching unit 52 
judges whether or not i f-the timer 50 is in operation or not . If the timer 50 is 
in operation, the watching unit 52 waits until stopping automatically. Then, 
15 the reproducing unit 56, after the first ON reproducing step S506, the OFF 
reproducing step S507, and the second ON reproducing step S508, reproduces 
the ON-OFF'ON operation of the button of the pointing device 44 14-so as to be 
recognized as a double click in the host PC 30. 

If judged not to be a double click at step S504, the process goes to the 
20 button operation judging step S509. 

At step S501, the data determining unit 54 checks whether or not if-the 
received operation packet shows the operation of changing the button of the 
pointing device 44 from ON to OFF or not . If Yes, the process goes to the 
second timer operation watching step S510. Otherwise, the process goes to the 
25 operation packet reproducing step S512 for reproducing the button operation. 

At the second timer watching step S510, the data determining unit 54 
judges whether or not if-the timer 50 is in operation or not . During operation, 
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waiting until stopping automatically, the process goes to the second timer start 
step S511. 

At step S511, the timer 50 starts, and the process goes to the operation 
packet reproducing step S512. 

At step S512, the reproducing unit 56 reproduces the operation that is 
indicated by the operation packet received at step S501. 

Thus, in the information processing unit 31, the data determining unit 
54 in the host PC 30 j udges the operation of the pointing device 44 in the client 
PC 40 by the data including the operation information that is sent from the 
10 client PC 40. The reproducing unit 56 reproduces the operation of the pointing 
device 44 in the client PC at the host PC 30 depending on the data that is 
judged by the data determining unit 54. 

According to the embodiment of the present invention, the double click 
is transmitted from the client computer to the host computer, not as the-plural 
15 operation packets showing an on/off operation of the button of the pointing 
device of the client computer , but instead as one operation packet showing the 
double click. Therefore, even if the delay time of the network is large, the host 
computer correctly recognizes the input of adouble click at the client computer. 

Moreover, the user of the client computer, by voice or display, is notified 
20 of the transmission of an operation packet showing a double click input from 
the client computer to the host compute r io noticed to tho uoor of tho client 
computer by voice or dioplay . Thus, the user instantly understands the input 
of a^double click. 

As described herein, in the remote control data processing method and 



system of the present invention, the information processing unit of the client 
computer converts the operating procedure of the pointing device of the client 
computer into operation information including button status data, coordinates 
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data, and interval data between operations, and transmits the operation 
information to the host computer. In the information processing unit of at-the 
host computer, the operating procedure of the client computer is reproduced 
from the received operation information. Thus, even if the delay time of the 
network is large, the host computer correctly recognizes the operation that is 
entered in the client computer. 

In the foregoing embodiment, the specific method and device are 
explained. The present invention may also be also realized by using a 
computer program product containing a computer program for executing the 
method shown in the embodiment by a computer. 
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ABSTRACT 

A system which provides for the remote control of ajiost computer by a 
client compute r io diocloood . The client computer converts the operating 
procedure of its pointing device into operation information including data 
regarding a about switch operation, coordinates data, and time interval data 
between operations, and transmits the operation information to the host 
computer. The host computer decomposes the received operation information 
into the original operating procedure, and reproduces the original operating 
procedure. Thus, the operating procedure of the pointing device that is 
transmitted to the host computer from the client computer is prevented from 
having the effects of an irregular delay of the network. 
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